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ABSTRACT 



Many universities currently encourage the use of online 
materials to enhance on-campus courses. Assessment of the impact of these 
technology- enhanced course initiatives is necessary to give insight into the 
best practices- -those that faculty should be encouraged to use based on 
student need. Assessment studies of this type will enrich the literature of 
distributed learning in higher education. In the fall term 1998, the 
University of Pittsburgh (Pennsylvania) conducted a pilot project to study 
the instructional impact and support needs for a Web-based instructional 
management system. The system centered on the "Courselnfo" software package 
acquired from Blackboard, Inc. The pilot was the result of a nine-month 
campus-wide software selection process, and a four-month effort to design and 
develop training and support systems. This paper describes the project, 
explains the assessment strategies designed for the pilot, and summarizes the 
results of formative and summative evaluation measures. The assessment 
includes data collected from students and faculty members in 20 courses (22 
sections) with 1,850 students. The paper includes 11 tables. (AEF) 
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Many universities currently encourage the use of online materials to enhance 
on-campus courses. Assessment of the impact of these technology-enhanced course 
initiatives is necessary if we are to gain insight into the best practices — those that 
faculty should be encouraged to use based on student need. Assessment studies of 
this type will enrich the literature of distributed learning in higher education. This 
paper will present and discuss 1 ) a process for designing assessment strategies to 
measure the impact of online course materials on students and 2) the results of 
formative and summative evaluation, including the categories of benefits reported by 
students. Based on a campus-wide initiative to train university faculty to use course 
management software, this assessment includes data collected during a pilot-term 
with over 1850 students enrolled in 20 courses (22 sections). 

Introduction 

In the Fall Term 1998, the University of Pittsburgh conducted a pilot project to study the 
instructional impact and support needs for a Web-based instructional management system. 
The system centered on the Courselnfo software package acquired from Blackboard, Inc. The 
pilot was the result of a nine-month campus-wide software selection process, and a 
four-month effort to design and develop training and support systems. 

This paper describes the project, explains the assessment strategies designed for the pilot, 
and summarizes the results of formative and summative evaluation measures. The 
assessment includes data collected from students and faculty members in 20 courses (22 
sections) with 1 ,850 student seats. 

Background 

The University of Pittsburgh is a state-related public research university and a member of the 
Association of American Universities (AAU). Founded in 1787, Pitt offers graduate, 
undergraduate, professional, and continuing education programs through 19 schools and four 
regional campuses. The University typically serves about 28,000 FTE in the fall term, and 
employs nearly 4,000 full- and part-time faculty members. Pitt's regional campuses are located 
between 30 and 170 miles from the main campus in Oakland, Pittsburgh’s cultural and 
medical center. Among the notable sites on the Oakland campus is the 42-story Cathedral of 
Learning with its 28 nationality classrooms, each representing the heritage of one of the 
region's many ethnic groups. 

The Center for Instructional Development & Distance Education (CIDDE) works directly with 
schools, departments, and individual faculty members to facilitate the academic goals of the 
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University in the areas of instructional development and technology, media support, faculty 
development, and distance education. With this broad mission of instructional development 
and support, CIDDE pursues new initiatives that will optimally support the academic priorities 
of the institution. These initiatives fall into three general categories: the acquisition or 
development of tools and facilities, the development and delivery of instructional support 
services, and the provision of assistance in the development of academic programs and 
instructional materials. 



During the summer of 1997, the Director of CIDDE met with 20 deans to identify their 
instructional technology and distance education needs and priorities. Based on these 
discussions she identified three common needs: (1) for computer-based conferencing, (2) for a 
method to easily create online materials, and (3) for a consistent "look and feel" to the 
University's online course offerings. 

Based on this identified need, a CIDDE instructional designer organized and chaired a 
committee to evaluate emerging software tools designed to facilitate the development and use 
of the World Wide Web for instruction. The committee included faculty and staff from CIDDE, 
the centralized computing services groups, and the University Library System, who 
volunteered to review existing software, and to try out selected packages to evaluate their 
potential utility. The committee articulated detailed requirements for a Web-based instructional 
package, identified the packages that addressed those needs, and ranked and rated each of 
the packages. The most promising five packages were selected for detailed evaluation, and 
trial copies were either accessed online or obtained, installed and tested by faculty in actual 
classes for the Spring Term 1998. 

In May 1999, after nine months deliberation, the committee recommended that the University 
acquire the Courselnfo instructional management system. The recommendation was 
presented, approved, and funded through the Software and Networked Information Working 
Group of the Executive Committee for Academic Computing. The work of the Web evaluation 
committee is summarized in a web,page at URL http://www.pitt.edu/~washburn/css.html . 

Courselnfo is an integrated software suite that enables faculty members to develop, manage, 
and use Web pages for instruction. Courselnfo acts as a "container" for organizing the 
instructional components of an online course. It offers tools to administer, grade, and record 
quizzes, to conduct synchronous chat sessions, and to hold asynchronous threaded 
discussion groups. It also provides user-friendly software to facilitate the instructor’s efforts to 
create new Web pages or incorporate existing pages, send e-mail, transfer files to and from 
students, and manage an online grade book. Courselnfo runs in both the Unix and NT Web 
server environments and is accessible using standard Web browsers and plug-ins. 



The primary factors driving the selection of Courselnfo included its ease of use and its 
adherence to standards such as SQL and the emerging Instructional Management System 
metadata modei (the EDUCAUSE IMS Project). Courselnfo was the only package previewed 
that enabled inexperience faculty members to create effective materials with no knowledge of 
HTML, yet still allow more advanced users to leverage their knowledge of HTML and 
Web-based authoring tools to produce sophisticated results. 



Implementation of the Pilot 



Project implementation was a cooperative effort between CIDDE and the centralized computer 
service units (the Computer Services and System Development unit and the Network Services 
unit), coordinated through a series of regular planning meetings and advised by several faculty 
committees. This section explains some of the important factors in the project implementation 
plan. 

The University acquired multiple licenses of the Courselnfo package and implemented three 
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servers in order to create a stable, secure and efficient environment. First, CIDDE 
implemented a test version of Courselnfo under Apache on a low-end Pentium processor 
running Linux. The test version was used to experiment with new releases and patches of the 
software. A second "development" server was implemented on a small Sun processor running 
Solaris and Apache. Faculty members used this server to develop their Web pages in a 
secure, confidential environment. The centralized Network Services group managed the third 
server, a high-end Sun processor also running Solaris and Apache. This "production" server 
was maintained seven days a week, 24 hours per day, and supported the online courses, 
faculty, and students. 

In response to concerns raised by several faculty committees about the sufficiency of remote 
access to the Pitt network, the Network Services group increased the dial-in modem pool by 
40% to over 800 dial-in ports. In addition, to handle the anticipated increase in student and 
faculty problems that might occur when implementing Web-based instruction, the Computer 
Services and Systems Development group expanded the existing single-shift help desk to a 7 
' 24 environment. The group also took proactive steps to prevent problems by ensuring that all 
requisite plug-ins were available in the public student computing labs and providing in-class 
orientations to students at the beginning of the term. 

Another potential issue was the anticipated increased desire by instructors to use computer 
projection technology in the classroom to demonstrate components of Courselnfo to their 
students. In response to this issue, the Classroom Management Team escalated the rate of 
classroom renovations to install computer projection devices in more classrooms. Cl DDE’s 
Instructional Media Services area also increased their inventory of portable microcomputers 
and projection units for mobile classroom presentations, and hired and trained additional 
student AV operators. 

To increase the likelihood of the project’s success, administrators in the Provost’s office gave 
two important directives. First, they made a strong commitment to training faculty in the use of 
Courselnfo, and second, they limited the number of courses that would be offered in the pilot 
term. For the Fall Term 1998 (99-1) pilot, 20 faculty members developed 20 courses (22 
different course sections) serving 1850 students. The goals of the pilot were to evaluate 
Courselnfo and to determine the access and support needs of implementing Web-enhanced 
instruction. 

Summer Instructional Development Institute 

Training for the pilot term was supported under the auspices of a newly proposed Summer 
Instructional Development Institute (SIDI). SIDI was initially conceived to provide intensive 
training and development opportunities for faculty over the summer. For its inaugural season, 
the Institute focused on training two groups of 12 faculty members in the use of Courselnfo. 
The first group was selected by the chairpersons of five science departments identified by the 
dean of the College of Arts and Sciences. This selection specifically targeted instructors of 
large, predominately freshmen courses with a goal of evaluating the effectiveness of 
Web-based course components in improving the experiences of students in large lecture 
settings (greater than 100 students per class). The second group was selected by the deans 
of schools and campuses that had expressed strong interest in the project. The faculty 
members involved were from the Faculty of Arts and Sciences (the Biology, Chemistry, 
Psychology, Geology, and Physics departments), the School of Education, the School of 
Nursing, the School of Pharmacy, the School of Health and Rehabilitative Sciences, and from 
the Johnstown and Bradford regional campuses. 

The Institute covered basic course design topics and featured hands-on instruction related to 
the mechanics of using Courselnfo and other related software tools. The courses were 
designed and taught in collaboration with the centralized computing services and library 
groups. 
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After analyzing Courselnfo and its anticipated training needs, several key decisions were 
made. First, training for Courselnfo would be implemented using Courselnfo. That is, faculty 
members would be enrolled in a Courselnfo "meta course" that would (1) serve as an 
exemplar of desired Courselnfo organizational structure and concepts, (2) serve as a 
repository of training and reference material, and (3) provide faculty with the opportunity to 
gain experience with Courselnfo from the students’ perspective. 

The second decision was to focus Courselnfo training more on the instructional design 
requirements of creating a Web course than on the mechanics of using the software package. 
Training would therefore promote the use of a straightforward instructional design model, 
originally developed at Pitt’s Learning Research and Development Center and later tailored to 
meet the needs of the University’s paper-based distance education program. This approach 
was possible because of the relatively easy task of training instructors in the use of 
Courselnfo, due to its friendly user interface. 

The final decision was to provide faculty course developers with a recommended structure in 
the form of a "template" course. In this approach, Courselnfo is viewed as a "container" of 
instructional materials. These materials are contained in documents and folders under 
appropriate subdivisions within a Courselnfo course site. Native Courselnfo ("out of the box") 
gives instructors a blank slate to contain any type of document in any structure or order. 
CIDDE instructional designers developed the "Pitt Courselnfo Template" to provide a 
more-specific structure to Courselnfo courses. This structure is intended to provide 
consistency across different courses, allowing students familiar with one Courselnfo section to 
easily adapt to a second. 

The Pitt Courselnfo Template specifies and explains a suggested structure for the "Course 
Information" and "Course Documents" components of Courselnfo. The Course Information 
component is considered analogous to a course syllabus and defined to include the following 
sections (documents): 

■ Introduction 

• Class Meeting Times 

■ Course Description 

■ Course Rationale 

■ Course Goals 

• Course Outline 

■ Course Materials 

• Course Requirements and Grading 

■ Course Policies 

■ Course Schedule 



Whereas the Course Information component of Courselnfo contains information about the 
course, the Course Documents component contains the course materials themselves. The 
template suggests that the course materials be subdivided into manageable units of work. 
These units, or modules, could be associated with chapters of a text, weeks of the term, or 
concepts of the course. Each module would be implemented via a separate folder within the 
Course Document component of Courselnfo. Each folder is defined to include the following 
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documents: 

• Introduction 

• Learning Objectives 

• Lecture Notes 

• Handouts 

• Exercises 

• Quizzes or Sample Tests 

• Related Readings 

The student would complete the course by working through each module in turn until all 
modules where mastered. 

Courselnfo training was designed around the process of implementing a Web-based course 
using the Pitt Courselnfo Template. For the Summer Institute, a five-day modular training 
program was developed that covered this process and addressed other related tools and 
issues. This five-day sequence was later divided into more discreet components. A one-day 
core course was a prerequisite for participation in Courselnfo. The core course, titled "Course 
Design And Development Using Courselnfo ," addressed the following topics: 

■ Introduction 

• Basic Web concepts 

• Courselnfo from the student’s perspective 

• Instructional Design Model and Process 

• Courselnfo Development 

■ Content editors 

/ 

■ Administration and course management 

■ Testing and grade book functions 

In addition to the required core training, several supplemental training modules were 
developed and offered to faculty. Participation in these modules was optional, base on each 
faculty member’s goals and interests. The supplemental modules included: 

• Microsoft Word and Virus Prevention Using Courselnfo 

• Microsoft PowerPoint with Courselnfo 

■ Designing and Acquiring Images for Courselnfo 

■ Using Audio and Video with Courselnfo 

■ Copyright and Legal Issues 
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Upon the completion of the Summer Institute, the pilot program was launched in the Fall Term 
1998. Data from subsequent terms reflect significant growth in the faculty’s adoption of 
Courselnfo (See Table 1 , page *). 

Formative Evaluation 



Evaluation of the pilot program was a crucial component of the project plan. To create an 
evaluation strategy, two instructional designers and a research design consultant formed a 
project assessment team. The team designed assessment goals and strategies based on the 
need to gain feedback from both students and faculty on issues related to access, support, 
and the instructional value of using Courselnfo. The evaluation team reviewed The Flashlight 
Project’s student inventory items as a basis of reference, drawing upon this pool of questions 
to develop an instrument to meet specific evaluation goals. 

Because the primary goal of formative evaluation was to identify any serious problems that 
students might encounter with accessing and using Courselnfo, early intervention was 
important. Therefore, in mid-October, CIDDE staff visited a representative sample of nine of 
the 22 course sections using Courselnfo and administered one-page paper surveys to all of 
the students attending class. Of the nine courses included in this evaluation, several were high 
enrollment, introductory science courses (the focus of the first Summer Institute). The sample 
also included undergraduate and graduate courses, low enrollment courses, and those offered 
on the Pitt’s main campus and regional campuses. Out of the 1850 students using Courselnfo 
during the pilot term, 485 students were surveyed for the formative evaluation. 

These data indicated that most students (92%) had already accessed the Courselnfo pages 
for their courses (by the sixth week of the term), and that more than half of them (59%) had 
not previously used a Web-based tool. The students indicated that the initial in-class 
explanations they received from the University computer support area or from their instructor 
on how to use Courselnfo were adequate (95%). Suggestions for improvements included 
providing handouts and conducting the training sessions in computer labs. Students reported 
very few difficulties using Courselnfo (87% of students surveyed did not contact the 24-hour 
help desk with questions/problems), and those who did have difficulty were able to obtain the 
needed assistance. Data from the help desk indicated that most student questions focused on 
how to configure their personal computers for dial-in access or how to establish computing 
accounts. 

One of the ten questions in the formative survey addressed the instructional value of 
Courselnfo, resulting in 69% of the students reporting that Courselnfo materials had helped 
their learning to a moderate or high degree. Additional positive student comments as to what 
Courselnfo components were helpful included "Lecture notes available online," "Checking my 
grades," "Assignments available online," and "Taking quizzes and practice exams." Only 4% of 
the students identified problems or concerns about using Courseinfo, specifying problems 
such as "access from home using a modem" and insufficient "computer lab hours." 

Summative Evaluation 

One of the main goals of this initiative was using instructional technology to enhance teaching 
and learning. Consequently, the summative evaluation focused on the instructional value of 
Courselnfo. The primary evaluation objective was to identify which components were helpful to 
students’ learning and why. 

Evaluation measures used during the pilot program included surveys, anecdotal data, help 
desk reports, and Web server statistics. Similar to the formative evaluation procedures, staff 
administered paper surveys to students in classrooms during the last two weeks of the term. 

Of the 1850 students enrolled in these courses, 796 surveys were returned. 
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The data indicated that students were accessing their course Web pages primarily through 
University student labs (436 responses), residence halls (125 responses), and via modem 
(263 responses). See Table 2, page * for Web server access statistics. 

Components Identified as Helpful by Students and Why 

Students were asked to identify which components were helpful (of those used in their 
courses) and why. A content analysis of students’ open-ended responses was completed, and 
the following summarizes these data. It should be noted that not all Course Info components 
were used in every course. For example, some courses did not use group work tools at all. 

The number of students’ open-ended comments for each item is reported in Table 3 through 
Table 9. 

Three of the most highly used components were also the ones that students rated as most 
helpful: Lecture Notes, Quizzes/Practice Exams, and Course Announcements (see Table 3, 
Page *). Other components rated as helpful included: Assignments, External Links, and 
E-maiL Student responses were veiy low for the remaining components (Group Work, 

Students Drop Box, Online Discussions, Surveys and Online Chats), probably because 
instructors used them only minimally during the pilot term. 

Lecture Notes in Courselnfo took a variety of forms, including basic lecture outlines, detailed 
instructor notes, and PowerPoint slides. Students listed these as a major asset to their 
learning. The majority of students responding to this question indicated that notes enabled 
them to better organize information before class, allowing them to spend more time listening to 
the lecture. Students also reported that the notes helped them clarity and organize the 
information they received in class. Many students reported using the lecture notes for study 
and review of class information and preparing for exams (See Table 4, page *). 

As expected, students felt that Quizzes and Practice Exams helped them to prepare for their 
in-class supervised exams and focus their study efforts (see Table 5 page *). Although these 
responses may reflect the availability of quizzes and practice exams, per se, and not the 
accessibility of them through Courselnfo, the features of immediate scoring and feedback 
through Courselnfo were clearly valuable to students. 

Predictably, students commented that Course Announcements were helpful for keeping them 
informed of course-related events, such as assignment due dates, information about 
up-coming tests, and any changes in the course procedures. Course Announcements was the 
most highly used component in Courselnfo (reported by faculty as used "extensively"), 
perhaps due to the ease of application by course instructors. Also, students unable to attend 
classes noted that Course Announcements helped them to keep informed (see Table 6, page 

!)• 

Students responded positively concerning the ease of access to assignments through 
Courselnfo (see Table 7 page *). The instructional value of the practice gained from these 
assignments rated high as welf^ with students commenting that they had a better 
understanding of what was happening in class by completing these assignments. As with 
Quizzes and Practice Exams, it should be noted that the students’ responses could be 
addressing the assignments, and not necessarily the added value of accessing them online. 
Many students explicitly commented on the usefulness of having these assignments online, 
however, as this procedure helped them to stay informed of due dates and follow a schedule 
for preparing and submitting assignments. 

Nearly two-thirds of the students commented on the ease of access that Courselnfo provided 
to additional and helpful information through External Links (see Table 8, page *). About 
one-third of students responding to this question specifically described that these links were 
helpful in completing homework and assignments. 
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Students were positive in their responses about e-mail communication (see Table 9 page *). 
Over two-thirds of the students responded that e-mail through Courselnfo enabled them to 
communicate more easily with their instructors--asking questions and conversing about the 
course. Almost 20% commented that the ease of communication through e-mail was helpful, 
while others reported that it helped in contacting other students and to keep informed. 

During the pilot term, instructors did not significantly use the Online Discussion, Group Work, 
Surveys, or Online Chat components of Courselnfo. Several instructors reported using these 
components only "minimally" and, therefore, these data will not be reported. 

Comparison of Courses 

Students were asked to compare the course they were currently taking, using Courselnfo, with 
other courses not using Courselnfo. In this comparison, students responded (either agreeing 
or disagreeing) that with Courselnfo they were more likely to perform 13 different 
instructionally-related activities. Of those students for whom these questions were applicable, 
Table 10 presents these data. The degree to which students reported using these activities 
varied from the highest rating of over 87% ("refer to course syllabus material") to a low of 58% 
("work on assignments with other students.") 

The two highest ratings were related to the access of online materials. Over 87% responded 
that they were more likely to "refer to course syllabus material," and 85% agreed that they 
were more likely to "access other online materials related to the content of this course" when 
compared with other courses not using Courselnfo. The next highest ratings related more to 
instructional activities and indicated that with Courselnfo, students were more likely to "review 
lecture notes to gain clarification," (85%), "complete assignments on time" (nearly 81%), "seek 
clarification when I did not understand something" (77%), and "actively participate in the 
course" (76%). 

Other preferred activities included receiving feedback on quizzes and exams quickly, 
communicating with the instructor, spending more time studying for the course, discussing 
ideas from this course with other students, discussing ideas from this course with the 
instructor, and working on assignments with other students 

Finally, students were also asked to agree or disagree with several statements as to the 
impact that certain online enhancements had on their learning. The highest positive response 
rate was to having Courselnfo used in other courses with over 90% agreeing. Almost 86% of 
the students agreed that online quizzes were helpful to their learning, and 81 % agreed that the 
online environment contributed to their learning in the course (see Table 1 1 , page *). 

Lessons Learned 

This section summarizes some of the lessons learned from the evaluations and experiences 
gained during the pilot and subsequent terms. The lessons are gleaned from an analysis of 
the formal formative and summative evaluation measures, Web server statistics, help desk 
reports, and anecdotal data collected from students, faculty, and support staff. 

First, using online enhancements increased the time that students were involved in learning 
tasks, such as accessing other online materials related to the content of the course, reviewing 
lecture notes to gain clarification, and completing assignments on time. Time on-task is one 
factor that has been found to contribute to increased student learning. Both students and 
faculty also valued the ability to use quizzes and practice exams, with immediate scoring and 
feedback. This was especially important in high enrollment courses where logistics and 
scoring time can be barriers to their use. 
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Students reported that they liked online materials and that the online environment contributed 
to their learning. Students clearly want these enhancements in their courses. Providing 
outlines of lecture notes prior to a class allows students to organize information before class 
and spend more time listening in class. Students reported that procedures as simple as 
providing lecture outlines have a major positive impact on their learning. It should be noted 
that few faculty reported any decrease in attendance as a result of using Courselnfo. 

During this pilot term, most instructors used more content-delivery components of Courselnfo 
(e.g., Course Announcements, Lecture Notes, and Course Assignments) than the interactive 
components (e.g., online discussions, chats, and group work), possibly because the former 
required less additional instructional design and development effort. However, as observed 
during terms subsequent to the pilot, faculty tend to develop online enhancements to their 
courses in stages, beginning with providing course materials online, then developing more 
interactive components, such as discussion and online quizzes. It is important to assist faculty 
to recognize the value of increasing the use of interactive components. As faculty become 
more familiar with these benefits, they may be more likely to design them into their courses. A 
recommendation for further study is to identify what changes, if any, are occurring with in-class 
teaching when online components are used. 

On the basis of the pilot and subsequent terms, the key ingredients of the success of the 
project include the ability to engage faculty, provide training and support, and plan and 
coordinate the project activities. 

Engage Faculty 

• Advice and consent : Involving faculty in the selection process for course 
management software is crucial. Moreover, collaborating on a campus-wide basis 
(Provost, Deans, Department Chairs, and support units) is an effective model. Few 
individual academic units have sufficient instructional and technical resources to 
effectively address all the needs and support for a project of this scope. The pilot 
program should be part of a broad-based strategic effort. 

• Motivated faculty: It is important to include motivated faculty in this type of pilot 
project. Those who are interested in using online materials will be more likely to 
create richer and more effective online materials, assist their colleagues in 
developing them, and be more tolerant of technology limitations and failures. 
Generally, they will become better role models for other faculty. For this reason, it is 
also important to seek and engage faculty influence leaders beginning in the early 
days of the project. 

Provide Training and Support 

• Require training: To use it effectively, faculty members need to ieam the capabiiities 
and idiosyncrasies of course management software. Faculty training should address 
course design concepts and introduce faculty to the software package and its local 
operational and support procedures. Requiring training will place less stress on 
support services in the future. At the University of Pittsburgh, a one-day core training 
module has been a required prerequisite for using Courselnfo for its four terms of use 
(over one year). In addition to training, on-going faculty support is critical to success, 
both in instructional design and in the use of technology. 

• Focus on instructional design : Include and emphasize instructional design in any 
training program. Faculty who work through the thinking/planning process of 
incorporating online enhancements into their on-campus courses use the 
components of Courselnfo to their fullest. Keep the emphasis in training on the 
functionality and instructional purpose of the technology, using the technology as a 
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tool and not as an end. 

• Adapt support services : Analyzing, reengineering, and implementing effective 
faculty and student support services is a critical success factor for Web-enabled 
instruction. These include services such as operating a 24-hour help desk, ensuring 
adequate remote access, providing standard browser and plug-in software in public 
student labs, providing in-class student orientations to the software, and supporting 
in-class AA/ presentations. 

Plan and Coordinate 

• Senior-level support : Because this project cut across multiple academic and service 
units, strong senior-level support was crucial to acquiring and implementing the tools 
and resources necessary to optimize the success of the project. 

• Communicate and coordinate : Involve related academic and technical support units 
and their key staff in planning and coordination activities, facilitated by group work 
technologies such as list servers, shared Web pages, and audio and video 
conferencing. 

• Follow-up : Plan and implement opportunities to discuss online instructional 
management systems among faculty who have used (and those who will be using) 
these tools with their courses. At Pitt, based on a recommendation from the Summer 
Instructional Development Institute faculty, a "Teaching with Technology Series" was 
successfully implemented. In the year following the pilot term, four faculty-led 
workshops were offered on topics such as "Creating a Web-based Lecture," 
"Internet-based Assignments that Work," and "Workable Online Discussions." 

• Evaluate : It is important to develop formative and summative evaluation plans and 
procedures for this type of technology-related course initiative. Although evaluation is 
generally included as part of any pilot program, it is necessary to design and carry out 
on-going assessment procedures. Such procedures help to gain insight into the best 
practices that faculty should be encouraged to use based on student need. 

Conclusion 

In summary, the pilot and subsequent terms demonstrate that, given a well-supported tool that 
enables them to use the Web for on-campus course enhancement, without requiring extensive 
technical knowledge, faculty will respond positively by implementing the tool, as illustrated by 
the dramatic growth in the use of Courselnfo at Pitt (See Table 1 ). 

Tables 

The following table is based on an analysis of hits (Web page visits) taken from the Web 
server log file generated by the production server for the Fall Term, 1998. 

Table 1: Courselnfo Growth by Term 





Fall 98 


Spring 99 


Summer 

99 


Fall 99 


Sections 


22 


115 


107 


350 


Seats 


1,850 


4,500 


2,000 


13,450 
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Table 2: Means of Access to Courselnfo 



Location 


Hits ! 


Percentage 


Residence Halls 


327,861 


19% 


Computing Labs 


543,643 


31% 


Regional 

Campuses 


132,171 


8% 


Other Pitt 


175,945 


10% 


Dial Up to Pitt 


203,921 


12% 


ISP (e.g., AOL, 
MSN) 


343,190 


20% 


Total 


1,726,731 


100% 



Table 3: Courselnfo Components Rated as "Helpful" (796 Responses) 



Component 


Responses 


Percentage 


Lecture Notes 


521 


66.6% 


Quizzes/Practice 

Exams 


331 


41.5% 


Course 

Announcements 


326 


41 .0% 


Assignments 


226 


28.4% 


External Links 


212 


26.6% 


E-mail 


180 


22.6% 



Table 4: Lecture Notes (475 Written Comments) 



45.9% | 


Helped me to prepare for class — 1 
printed notes before class and was able 
to listen more to the lecture; added 
organization to the lecture. 


24.4% 


Helped clarify class 
information — something 1 did not 
understand in class; helped me to 
organize my notes 


15.6% 


Helped with studying and reviewing for 
exams 


14.1% 


If 1 missed a lecture, 1 could get the 
basic information 



Table 5: Quizzes/Practice Exams (310 Written Comments) 
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83.9% 


Helped to prepare for exams and focus 
my studying efforts 


14.5% 


Practice was useful 


1 .6% 


Useful for grade posting 



Table 6: Course Announcements (268 Written Comments) 



89.6% 


Helped me to keep informed of 
course-related events 


10.4% 


If 1 was unable to attend a class, this 
helped me to stay informed 



Table 7: Assignments (169 Written Comments) 



49.7% 


Ease of access to assignments 


27.2% 


With the practice gained from 
assignments, 1 had better 
understanding. 


23.1% 


Easily stay informed of what was due; 
could follow a schedule 



Table 8: External Links (191 Written Comments) 



63.6% 


Provided easy access to external 
information; provided more and helpful 
information 


33.0% 


Provided help with homework and 
assignments j 


3.4% 


Provided help with research 



Table 9: E-mail Communication (144 Written Comments) 



67.4% 


Ease of contact with instructors and to 
ask questions 


18.1% 


Ease of communication 


9.7% 


Helped to keep informed 


4.9% 


Helped to contact other students 



Table 10: Preference for Courselnfo vs. Non-Courselnfo Courses 
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Activity 


Students 

Agreeing 


Refer to course syllabus. 


87.2% 


Access other online materials related 
to the content of this course. 


85.4% 


Review lecture notes to gain 
clarification. 


85.1% 


Complete assignments on time. 


80.8% 


Actively participate in the course. 


77.4% 


Seek clarification when 1 did not 
understand something. 


76.2% 


Receive instructor comments on 
assignments quickly. 


75.5% 


Receive feedback on quizzes and 
exams quickly. 


73.9% 


Communicate with my instructor. 


70.8% 


Spend more time studying for the 
course. 


67.4% 


Discuss ideas from this course with 
other students. 


62.0% 


Discuss ideas from this course with 
the instructor. 


59.8% 


Work on assignments with other 
students. 


57.7% 



Note: In the above table, the Agree" and "Strongly Agree" responses were combined, the "Disagree" 
and "Strongly Disagree" responses were combined, and the "Not Appropriate" responses were not 
counted. 



Table 11: Impact of Online Learning Components 



Statement 


Students j 
Agreeing j 


1 would like to have Courselnfo used j 
in other courses. 


90.2% 


Online quizzes were helpful to my 
learning. 


85.9% 


The online environment contributed 
to my learning in the course. 


81 .4% 


Online discussions encouraged me 
to think more about the course 
concepts than 1 usually do. 


53.8% 
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